Mixed Reality Annotation of Robotic-Assisted Surgery videos with real- time tracking and stereo matching

Robotic-Assisted Surgery (RAS) is beginning to unlock its potential. However, despite the latest advances in RAS, steep learning curve of RAS devices remains a problem. A common teaching resource surgery use videos previous procedures, which are almost always stereoscopic. It important be able add virtual annotations onto these so that certain elements surgical process tracked and highlighted during session. Including stereoscopic turns them into Mixed Reality (MR) experiences, tissues, tools procedures better observed. an MR-based annotation objects requires tracking some kind depth estimation. For this reason, paper proposes real-time hybrid tracking–matching method for performing on videos. The proposed because it combines stereo matching, avoiding need calculate real pixels. was tested with six different state-of-the-art trackers assessed sigmoidectomy sigma neoplasia, performed Da Vinci® X system. Objective assessment metrics proposed, presented calculated solutions. results show can successfully annotate real-time. Of all method, CSRT (Channel Spatial Reliability Tracking) tracker seems most reliable robust terms capabilities. In addition, absence absolute ground truth, domain expert using novel continuous-rating Oculus Quest 2 Virtual device performed, showing perception good, fact no values calculated.